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ABSTRACT 

This document examines issues and techniques related to 
assessment of online learning for adult, career, and vocational education. 

The document begins with an overview of the disadvantages of assessing at a 
distance and the issue of security in the online environment. The following 
aspects of continuous assessment are discussed: characteristics (careful, 
systematic, and planned) ; issues (pacing, feedback, and learning quality) ; 
and types of continuous assessment activities (ungraded activities and 
feedback built into study materials; self-assessment quizzes and tests; 
formal feedback on assignments from instructors, peers, or workplace 
colleagues or mentors; informal dialogues with instructors, peers, and 
others; and ungraded tests that prepare learners for graded assessment) . 
Presented next are guidelines for making online assessment interactive by 
taking actions such as the following: comparing learner answers with correct 
answers and giving feedback; recommending remediation; and showing Web pages 
with more information on missed questions. The following activities of 
effective instructors in online assessment environments are considered: 
acting as a coach, mentor, and facilitator; communicating information; 
providing timely feedback; and being a team member. The advantages, 
disadvantages, and examples of the following forms of assessment are 
detailed: case studies; discussions; microworlds/simulations ; electronic 
portfolios; and self-assessments. The bibliography contains 45 references. 
(MN) 
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More and more teaching and learning are taking place at a distance, especially on the World Wide Web. How 
do educators measure what distance learners know? This issue of the Practitioner File looks at issues and tech- 
niques related to the assessment of online learning for adult, career, and vocational education. We address 
assessment in the sense of obtaining and interpreting information about what has been learned, rather than 
evaluating web-based courses and instruction (although we found that the literature tilts more toward course 
evaluation than learner assessment). Inside you will find discussions of the principles, advantages, and chal- 
lenges of online assessment; the concepts of continuous assessment and interaaivity; the role of the instructor; 
and descriptions of types, tools, techniques, and resources. 



Online Assessment: Principles and Practices 



Learner assessment has a foundation of good practice 
whether it takes place online or face to face (Dirl^ 1997; 
Zvacek 1999). As.sessmem is usually intended to provide 
both instructors and learners with information on progress 
and to measure achievement of learning goals. Formative 
assessments provide ongoing feedback to improve instruc- 
tion and learning. Summative assessments are made to 
assign value to what has been learned (Australian National 
Training Authority 1999; Hopper 1998). Principles of good 
assessment include validity (does it measure what is in- 
tended?), reliability (docs it consistently produce the same 
information?), flexibility (are various methods and ap- 
proaches used to accommodate diverse needs?), and fair- 
ness (is it free of biases?) (ANTA 1999; Juchnowski and 
Atkins 1999). The quality of test construction matters: 
“Bad questions... lead to poor test results. Whether pre- 
sented to students on paper, electronically, or verbally, good 
questions are an essential feature of a test" (Sephton 1999, 
n.p.). Because assessment has multiple purposes, multiple 
methods are needed. 

These principles apply to assessment regardless of the de- 
livery mechanism. Tl^ere are, of course, differences between 
online and face-to-face situations. Some of the disadvan- 
tages and advantages to conducting assessment at a dis- 
tance are discussed here, concluding with some principles 
and practices for online assessment. 

Disadvantages of Assessing 
at a Distance 

"Assessment and measurement become even more critical 
in the absence of the face-to-facc interactions that enable 
teachers to use informal observation to gauge student re- 
sponse, obtain feedback, and progress toward goals” (Penn- 
sylvania State University 1998, n.p.). This lack of non- 
verbal cues poses the greatest challenge to online assess- 



3 



ment. Other disadvantages include the following (“As- 
sessing Students” 1 999: Juchnowski and Atkins 1999) — 

• Learner isolation and the impersonality of computer- 
ized assessment 

• Lack of instructor control over assessment conditions 

• Uneven access of learners to technology and resources 

• Technical problems associated with access— e.g., 
media-rich learning activities such as video segments 
may be expensive to produce and slow to download; 
failed hardware or phoneline outages may prevent 
access 

• Learners’ and instructors’ technical skill levels 

• Learner anxiety about being disadvantaged by 
difficulties with the technology 

• Instructor time to score tests end provide feedback 

• Fatigue — lengthy assessment activities may stress the 
eyes from the use of computer screens 

Two issues that may create particular barriers to online 
assessment are security and the need for programming. 

Security 

Ensuring that the individuals completing the assessment 
are who they say are (authentication) "is an issue in all 
examinations... However, in a computer environment, im- 
personation may be perceived as a greater risk” (Mogey 

continued on p. 8 



Online Assessment: Continuous &. 1 nteractive 2 

The Role of the Instructor. 4 

Types of Online Assessment 5 

Adult, C!areer, Voc Ed Resources 7 

Distance Assessment Principles 9 

Web Resources 10 

Bibliography. I 1 



i 







I 



I 

! 

i 

! 



Onl^e AssiBssinneiit: 

. Cantinuauis aiflil Interaictive 



Communication works somewhat differently in web-based 
training and distance education than in the traditional 
classroom. In the traditional classroom, face-to-face com- 
munication allows all panics to send and receive a full 
range of verbal messages, both written and oral, as well as 
nonverbal messages like facial expression, tone of voice, 
and inflection that give additional information. In par- 
ticular, traditional classroom instructors can monitor non- 
verbal communication as a check on learners* comprehen- 
sion; a puzzled look or a pause in notetaking can indicate 
that learners are not comprehending (Champagne 1 998). 

Online communication using computer* mediated commu- 
nication (CMC) mechanisms such as e-mail and computer 
conferencing is different in two important respects, how- 
ever (Bigelow 1999). First, online communication is not 
as rich; it's usually only written text with static graphic 
images (if any). Since instructors cannot observe learners, 
there are no nonverbal messages indicating comprehension 
or lack thereof. Second, online communication is typically 
slov/er because you can't type as fast as you can talk. The 
extra time allowed by that slower rate of communication 
can lead to more thoughtful interactions, but it can also 
have an adverse effect on the interpersonal interaction. 

Continuous Assessment 

Those important differences in communication mean that 
in distance education, assessment for formative purposes 
becomes much more important. Since online instructors 
and learners cannot comnaunicate as quickly, easily, and 
fully, instructors need to take extra steps to monitor learn- 
ers' performance and comprehension (Lindner 1998). Con- 
tinuous assessment allows instructors to familiarize them- 
selves vrith each learner's work repeatedly to help ensure 
that understanding occurs (Schrum 1998). In other words, 
since online instructors can't see their learners, can't ob- 
serve nonverbal clues, and can’t ask quick, impromptu 
questions to check comprehension, instructors must use 
other means to ensure that learners are on the right track 
(Thorpe 1998). 

Careful, Systematic, and Planned 

The results of continuous assessment can certainly be graded 
and included in determining a final course grade, but the 
purpose is more formative than summative. So to be effec- 
tive, continuous assessment must be more than just a final 
add-on to instruction for determining a grade; it must be 
planned carefully and systematically as an ongoing part 



of instruction. Planning for continuous assessment should 
take three key issues into account (ibid.): 

• Pacing. Continuous assessment should require that 
learners demonstrate and use whaf they have learned by 
corrtpleting an independent piece of work (alone or in 
groups) at a specified point in instruction. Tlie mechanis- 
tic constraint of pacing helps learners manage and sched- 
ule their own learning, v/hile it also enables constructivist 
learning by providing scaffolding that helps learners use 
what they have already learned to construct a coherent 
understanding of new information. 

• Feedback. Instructors provide feedback on the quality 
of learners' understanding. Again, feedback provides ex- 
pert scaffolding to help learners construct understanding; 
timely, critical feedback helps learners improve and refine 
their early understanding. In addition, positive feedback 
provides experiences of success that motivate learners to 
continue, overcoming obstacles if necessary. 

• Learning quality. Both pacing and feedback directly 
affect whether learners study and learn and how effectively 
they do so. The assigned activities determine what learn- 
ers learn by directing them to specific content areas and 
by requiring specific skills (e.g., problem solving, calcula- 
tion, analysis, synthesis, communication). 

Kinds of Activities 

A range of activities can be used for continuous assessment 
to help develop learning, as long as assessment activities 
are closely aligned and interwoven with study materials 
(Morgan and O’Reilly 1999): 

• Ungraded activities and feedback built into study 
materials 

• Self-assessment quizzes and tests that allow learners 
to check their own learning 

• Formal feedback on assignments from instructors, 
peers, or workplace co.Ueagues or mentors 

• Informal dialogue with instructors, peers, or others 

• Ungraded tests that prepare learners for formal , 
graded assessment 

Assignments and informal dialogue offer many opportuni- 
ties for continuous assessment in particular. For example, 
learners can be required to participate in online CMC con- 
ferences; they might be required to submit a minimum 
number of messages contributing to the discussion and re- 
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ceive instnictor feedback on the quality of those messages; 
or they might receive feedback on the quality of specific 
contributions they submit for feedback (Thorpe 1 998). As- 
signed activities and dialogue can also involve group ac- 
tivities or projects in which learners collaborate to solve a 
problem, create a simulation, design a produa, or com- 
plete anask (Schrum 1998). Of course, one critical crite- 
rion in identifying assessment activities is that they must 
relate to the learning objectives you have established 
(Jarmon 1999). 

Pravidiny Feedback 

The feedback instructors give in continuous assessment 
should provide the scaffolding that allows learners’ un- 
derstanding to grow and evolve. It should be full and spe- 
cific enough to tell learners clearly when and how they are 
on track or off track~for example, “Good job identifying 
the primary differences. .. [between two theories]. Your re- 
sponse is right on target. Should this be an exam question, 
you might want to elaborate a bit on each difference (e.g., 
give an example of each of these differences related to a 
teaching strategy)" (Hazari and Schnorr 1999, p. 36), Inex- 
perienced online learners may be particularly reassured by 
instructor feedback that confirms they’re on track and doing 
well (Thorpe 1998). 

Feedback should be timely and prompt: full, specific feed- 
back is most useful when learners receive it in time to use 
it effectively. Furthermore, learners arc most likely to use 
feedback productively when they arc oriented beforehand 
to its purpose and significance, and when feedback is pro- 
vided in a consistent format (e.g., separate comments on 
stnuaure and format of learner work; sequence of comments 
from the genera! to the specific) (Mann 1998). Finally, 
feedback — particularly on graded assessments — should al- 
ways be based on consistent, clearly communicated stan- 
dards, or rubrics (Nelson 1998), 

Interactive Assessment 

Online assessment in distance education can make use of 
Internet technologies that allow greatly increased commu- 
nication, interactivity, and collaboration compared to the 
traditional classroom or paper-and-pencil forms of distance 
education. CMC allows synchronous and asynchronous, 
one-to-one, one-to-many, and many-to-many communica- 
tion; its use of text-based communication promotes thought- 
ful and reflective commentary (Schrum 1 998). The inter- 
activity possible in online continuous assessment, either 



with the online environment or with instructors or other 
learners, helps engage learners in active application of 
knowledge, principles, and values and provides them with 
feedback that allows their understanding to develop (Hazari 
and Schnorr 1 999). Hence, online assessment can play an 
important pan in a constructivist learning experience. 

In addition, online assessment can take advantage of the 
capabilities of other software applications to develop in- 
teractive online tests with a wide array of functions 
(Dickinson 1997): 

• Record answers to a file the instructor can review 
later. 

• Compare learner answers with correct answers and 

give feedback. — 

• Grade the test and give feedback. 

• Recommend remediation. 

• Show the learner a web page with more information 
on a missed question. 

• Prevent the learner from proceeding to new instruc- 
tion until the learner passes the test. 

• Control retries — allow any number or a restricted 
number only. 

• Require the learner to enter a name and controlled 
password. 

• Restria the dates on which learners can take the 
test, avoiding last-minute test-taking. 

• Time the test. 

• E-mail test results to the instruaor. 

Developers can create customized online tests from scratch 
using programming languages such as Common Gatew'ay 
Interface (CGI) (ibid.), Active Server Pages (ASP), and 
JavaScript (Doulai and Stace 1998; Gray 1999) or com- 
mercially available packages such as Toolbook II, 
Macromedia Authorware, Jamba, or CGI*Star (Dickinson 
1997). Easiest to use are commercially available template- 
based instructional management systems, for example, 
WebCT, Blackboard, TopClass, and Web-Coursc-in-a-Box 
(Bigelow 1999). Interactive tests can include multiple 
choice, true/false, matching, and even essay questions 
(ibid.), although scoring essay questions is difficult for 
anything beyond merely looking for specific words Or se- 
quences of words in learner answers (Dickinson 1997). 



The Role of the Iimstructor 
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A key factor that distinguishes distance education from 
the traditional classroom is the dramatic shift in the role 
of the instructor from being the authoritative source of 
knowledge to being a facilitator of learner-centered, col- 
laborative, and egalitarian learning (Hutton 1999). In- 
stead, instructors can take advantage of the potentially 
constructivist online environment to build in opportuni- 
ties for creativity, so that learners can relate learning to 
their experience and local context. The governing principle 
is that learning is an active process in which learners con- 
struct rather than acquire knowledge, and the instructor's 
role is to support that construction rather than communi- 
cate knowledge (Milman 1999). 



Being a Mentor, Coach^ and Facilitator 

That constructivist perspective also applies to the 
instruaor's role in online assessment. First, instruaors may 
find that they do‘to questioning, probing, and focusing 
in continuous assessment. In the online environment, the 
capabilities of CMC allow learners to take a more active 
and facilitative role. The instructor assumes a more social 
role: forming a supportive learning environment, promot- 
ing human relationships, maintaining the group as a unit, 
and helping members work together in a mutual cause 
(Hutton 1999). In addition, distance education students 
are often typical adult learners — self-directed, motivated 
by personal learning goals, willing and able to apply their 
own experience to learning content (Tillson et al. 1998). 
As a result, the instructor’s role as mentor, coach, and fa- 
cilitator helps encourage connections and learning made 
possible by carefully, systematically planned continuous 
assessment (Elliott 1999). 



Furthermore, instructors arc often tl\e first point of con- 
tact for distance education learners when they need ad- 
ministrative, technological, or affective support (ibid.). 
Instructors must be able to provide referrals and linkages 
to technology services, support services, and other institu- 
tional resources. Inexperienced learners often rely on assis- 
tance from instructors to acquire the technology skills 
needed to condua a web search or participate in a chatroom 
discussion (ibid,). And prompt intervention with affective 
support and even tips on stress management is well appre- 
ciated by distance learners who are uncertain how to pro- 
ceed: 

Cammunicating Infarmatian 



In the traditional classroom, instruaors are used to pro- 
viding direction, guidance, and feedback orally as a part 
of assessment. In online assessment, however, instruaors 
must provide that same dircaion, guidance, and feedback 
in writing; thus, instructors must communicate clearly and 
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effectively in a different medium (ibid.). Instructors are 
likely to find that writing takes more time than talking, 
and they may also find that, like learners (Hutton 1 999), 
they mnt to take more time and be more thoughtful when 
communicating in writing. 

One key raaor in communicating effectively in writing is 
the content of feedback provided to distance learners. The 
content of feedback needed can range from a simple “Yes, 
that’s correa” to a much more complex presentation in a 
whole new way to aid understanding (Gibson et al. 1 995 ) 
Similarly, distance learners have a particular need to know 
dearly and spedfically what the expectations are and what 
constitutes successful outcomes; they should know this bt • 
fore they begin the course. Working on the Web may re- 
quire more detailed descriptions and definitions of success 
(Hardy 1999), Weight assigned to different factors in 
graded assessments (e.g., 30 percent for presentation and 
organization, 70 percent for content and relevance) is es- 
pecially important (Mann 1998). Learners typically make 
more positive comments on instructor feedback on actual 
assignments when that feedback is fuller and more detailed 
(Thorpe 1998). 

Instructors should also remember that online communica- 
tion involves more than feedback on the details of learning 
content (ibid.). Online learners also value advice and guid- 
ance to help them start and stay on track. Inexperienced 
learners may especially appreciate confirmation that they 
are on track — they may be glad to have that specific reas- 
surance even though their instructor has said that he or 
she would step in if they got off track. 

Praviding Timely Feedback 

The timeliness of feedback instructors provide is crucial. 
There is little point in making extensive comments on 
learner work if the learner doesn't receive them before going 
on to the next assignment; the learner needs to be able to 
incorporate the feedback into the next assignment (Dirks 
1 998). Learners usually react favorably to frequent con- 
tact and feedback from instructors (Tillson et al. 1998), 
especially in the initial planning stages of project work. 

Although the asynchronous nature of CMC is usually one 
of its most appealing features, learners may benefit from 
online office hours at specified ti mes so learners can log on 
and talk with the instructor in a chatroom (ZirkJe and 
Ourand 1999). Instruaors can monitor the effcaiveness 
of their online communication by asking learners about 
their perceptions and assessment of the quality of that feed- 
back (level, structure, content, relevance, usefulness) in 
order to improve effectiveness (Mann 1998). 
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Being a Team Member 

Instructors must keep in mind that they may be only one 
member of the distance learning team, which can also in- 
clude instructional designers, instructional developers, 
technology supporters, facilities supporters, materials sup- 
porters, distant-site facilitators, and management sponsors 
(Abernathy 1 998), As team members, instruaors often find 
that far more time and attention to detail are required 
than for face-to-face courses. Teamwork and cooperation 
among instructors, technical media specialists, and admin- 
istrators are critical. Instruaors may need to complete 
course plans and materials well in advance so that other 
team mcnibers can do their part (Tillson et al. 1 998). 

Equally, instructors often receive significant support and 
assistance from other team members, particularly technol- 
ogy staff (McDougall, Place, and Currie 1999). For ex- 
ample, technology support staff may provide online man- 
agement systems designed to enable instructors with little 
or no computer experience to create online tests with mul- 
tiple choice, short answer, and essay questions incorporat- 
ing formatted text, images, video, and audio both for learner 
self-assessment and graded assessment. Systems might ran- 
domly create tests from item data banks, grade the results, 
and post them both to learners and instructors. 

Such sophisticated online systems often make their own 
demands on instruaors. If an online system creates ran- 
domized tests using an item bank, instructors may need to 
write additional test items from which to draw. Likewise, 
if the online system provides content- related hints for learn- 
ers during the test, instructors need to provide that con- 
tent information to technology staff. Finally, instructors 
may need to supply detailed content information in the 
forms of hints (received by learners during the lest, based 
on their answers to guide them in the correct direction) 
and key words or other forms of expert answers (Kennedy 
and Eizenberg 1998). 

Institutional technology support might offer professional 
development opportunities like training sessions in spe- 
cific technologies, long- or short-term programs that pro- 
vide the time and resources for instructors to acquire tech- 
nology skills, targeted workshops and demonstrations, and 
one-on-one consultation. Assistance may be available not 
only from full-time technical experts but also from part- 
time student staff who are familiar with software applica- 
tions. Instruaors can also develop skills by sharing experi- 
ences, ideas, and discoveries with one another (Zirkle and 
Ourand 1999). 



A variety of methods may be used online to assess learn- 
ers in the following areas (Morgan and O’Reilly 1 999): 

• Critical thinking (essays, reports, reflective jour- 
nals) 

• Problem solving (multimedia or text-based sce- 
narios, simulations using CD-ROM, videocon- 
ferencing) 

• Demonstrating techniques (videoconferencing, 
verification by workplace mentor, site monitor) 

• Self- management (journal, autobiography, 
portfolio, learning contract) 

• Information access/managcmcnt (database devel- 
opment, bibliography, problem solving) 

• Demonstrating knowledge (written exam with local 
proctors, quick feedback through multiple choice, 
true/false matching, short answer tests) 

• Designing, creating (portfolios, projeas using 
video or the Web) 

• Communicating (debate, role play, PowerPoint 
presentation, report journal, essay) 

• Teamwork and collaboration (e-mail, listserv, or 
conferencing discussions/debates) 

Advantages, disadvantages, and examples of several 

types are presented in the chart on page 6. ^ 






Advantages 



Disadvantages 



Examples 



Case Studies 
CDirks 1998, 
Nicdcrhaiiser 
ctal.1999) 



Current, authentic, 
-quire compre- 
hensive appiication 
of learning 



Subj calve, time 
consuming for learners 
to complete and 
i nsiruaors to grade 



Discussions 
(Nelson 1998; 
Tarouco, de Vit. 
Hack, and 
Gdlcr 2000) 



Require thoughtful 
conceptualization 
and presentation of 
ideas; encourage 
some who are 
intimidated by face- 
to-face discussion; 
instnetor monit- 
oring of discussions 
catches problems 
early 



Generate huge amounts 
of text to be evaluated, 
may require new types 
of assessment criteria, 
present time and access 
constraints 



Microworlds' 
Simulations 
(Hopper 1998; 
Sdtaacr etal. 
1999) 


Require learners to 
consti*ua know- 
ledge and use 
mctaoogniu’vc 
strategics; allow 
performance-based 
assessment 


Can involve complex 
programming and 
specific hardware and 
software 


Electronic 


Accommodate 


Can emphasize 


Portfolios 


multiple 


bcllsAvhistIcs; require 


(Aschermann 


intelligences; 


time to compile and 


1999; Milman 


present a cross- 


assess; aeators and 


1999) 


seaion of achieve- 
ments and skills; 
capture performance 
data; require critical 
scL‘^-aSScssment 


assessors need tech- 
nical skills; storage 
space and trans- 
portability may be 
barriers 


Self- 


Teach a lifelong 


Must account for 


Assessments 


skill; important in 


diverse backgrounds 


(Bigelow 1999; 


distance learning 


and approaches to 


Taylor 1998) 

IS 


because of potential 
remoteness, 
isolation, few 
opportunities for 
interaaion and 
monitoring; online 
instmments are 
easily scored and 
analyzed, providing 
instant feedback; 
learners can retry 
and reevaluate 


study: self-assessment 
that provides only 
model answers or 
solutions is not useful 



Four interactive ease studies on disease prevention and health 
promotion (Niederhauser ct al. 1999) were included in a nurse 
practitioner web-based course using two types of CGIs-NetCloak and 
Nctfomis. The site induded links to faculty, a syllabus, bibliography, 
guidelines for task completion, required readings, grading criteria, and 
the evaluation. The tasks induded finding websites relevant to eases, 
answering questions about eases online, and partidpating in online 
discussions over 2 weeks. Learners were evaluated on their use of the 
Internet for learning as well as ease discussions, answers, and 
websites found. The eases thus developed both content knowledge 
and information literacy skills. 



Tarouco ct ai. (2000) describe a learning environment developed to 
support group work and learner assessment tlirough the Internet 
using a set of computer-mediated distance learning tools: a consensus 
tool (based on Delphi technique) collects and tabulates responses and 
level of contributions to discussions; a tracking tool registers pages 
accessed, datc/time; a voting tool provides fast feedback from learners 
to instniaor, and a self-evaluation tool enables learners to chart their 
own progress. 



Middle school students (Schaaer et al. 1999) move from novice to 
expert knowledge of environmental sdcncc by assembling a concept 
map, an authentic task involving electronic infonmatioit-sccldng sWlls 
(exploration, aeation, relcvana. cxiraaion, synthesis, organizing, 
representation). Rxjr computer tools were used: (1) Java Mapper 
(concept mapping software); (2) a simulated web environment 
(database of 200-f selcacd websites); (3) a bookmarking Java applet; 
and (4) feedback provided in real time (comparison of learners' 
performance to expert performance). Students could also construa 
knowledge maps synchronously and collaborativeiy using Hyper 
C^id* orjava Imowledgc management sof^vare. 



Electronic portfolios are being used inacasingly in preservioe teacher 
education. Aschermann (1999) desaibes how preservice teachers 
originally created them on CD-ROM, but they were difficult to 
update and production hardwarc/software was costly. They switched to 
Netscape Composer, a free program requiring little technical 
knowledge. Use of the Web improved portfolio quality and allowed 
assessment of technology skills as well. 



ScIfTcst <httpyAvww.usq.cdu.au/uscrs/tayloqa/Scl ftest.htm> is an 
Australian lastrumcnt designed so that learners can write out their 
solutions to math problems in detail, have a means of comparing their 
solution with a model solution, have a number of alternative model 
solutions available to them if appropriate, be credited when they get 
only part of a solution correct, and be given a summary at the end of 
cadt session detailing which topics they still have to master. 
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staff. 

Canale, R., and Duwart, E. “Internet- 
based Reflective Learning for Coop- 
erative Education Students during 
Co-op Work Periods." Journal of Co- 
operative Education 34, no. 2 (1999): 
25-34. 

Explains how distance learning links 
cooperative education students to 
their institution during work experi- 
ences. Describes the use of confer- 
encing software for assignments and 
reflective discussions. 

Gallant, G. M. “Profes,sional Develop- 
ment for Web-based Teaching: Over- 
coming Innocence and Resistance." 
New Directions for Adult and Continu- 
ing Education no. 88 (Winter 2000) 
forthcoming, 

Addresses faculty development issues 
and lists numerous resources to help 
prepare teachers for web-based in- 
struction, 

Hayes, 1C., and Huckstadt, A. “Devel- 
oping Interactive Continuing Educa- 
tion on the Web "Journal of Continu- 
ing Education in Nuning 3 1 , no. 5 
(Septembcr-Oaobcr20(X)): 199-203. 



Explores the u.se of Java scriptlets to 
provide formative evaluation and 
feedback throughout virtual patient 
scenarios in online modules for nurse 
practitioner self-study. 

Human, S. E.; IGlboume, L. M.; Clark, 
T. D.; Shriberg, A.; and Cunning- 
ham, B. “Using Web-Enhanced In- 
struaion in an Interpersonal Skills 
Course." Journal of Management Edu- 
cation 23, no. 5 (October 1999): 584- 
606. (EJ 592 565) 

Describes how management students 
developed criteria they then used to 
select web-based diagnostic self-assess- 
ment tools for a course on interper- 
sonal skills. 



Journal of Management Education 23, 
no. 6 (December 1999). Special Is- 
sue: Innovative Uses of Information 
Technology in Management Educa- 
tion. 

Bigelow, I. D. “The Web as an Orga- 
nizational Behavior Learning Me- 
dium," pp. 635-650. (EJ 596 509) 
Addresses the impact of moving learn- 
ing activities to an online medium, 
including effects on team projects, role 
plays, self-assessment, interactive cases 
and simulations, exams, and feedback. 

Chappell, D, S., and Schermerhorn, 
J. R., Jr “Using Electronic Student 
Portfolios in Management Education: 
A Stakeholder Perspective." pp. 65 1 - 
662. (EJ 596 510)“ 

Discusses how a business school is us- 
ing electronic student portfolios for 
academic assessment, career develop- 
ment, internship, and job placements. 
Recommends that they be mandatory, 
even for students with weaker com- 
puter skills, and they should have 
defined deadlines and feedback 
mechanisms. 

Shrivastava, P. “Management Classes 
as Online Learning Communities." 
pp. 691-702. (EJ 596 512) 

Describes the use of eSocrates software 
for networked communications in the 
aeation of an online learning commu- 
nity in business education. 



Meiscl, S., and Marx, B. “Screen to 
Screen versus Face to Face; Experienc- 
ing the Differences in Management 
Education." pp. 719-731. (EJ 596 
514) 

Presents an online experiential exer- 
cise on power and negotiation to ex- 
plore differences between online and 
traditional education. 



Niederhauser, V P.; Bigley, M. B.; Hale, 
J.; and Harper, D. “Cybercases: An 
Innovation in Internet Education.” 
Journal of Nursing Education 38, no. 
9 (December 1999): 415-418. (EJ 
596 594) 

Describes a mix of traditional and 
technological modes to present online 
case studies for nurse practitioner con- 
tinuing education. Addresses security, 
computer literacy, and evaluation Is- 
sues. 

Schilke, R. A. “Web-based Instruaion 
for Adult Educators — Hi-Touch ver- 
sus Hi-Tech." \nProceeding;5of the 19th 
Annual Midwest Research to Practice 
Conference in Adults Continuing, and 
Community Education, edited by M, 
Glowacki-Dudka, pp. 208-212. 
Madison: University of Wisconsin, 
2000 . 

Asserts that most literature and prac- 
tice in web-based instruction is behav- 
iorist. Attempts to identify course de- 
signs and techniques that are better 
aligned with adult learning principles 
and theories. 

Zirkle, C., and Ourand, D. B. “Teach- 
ing a Course through Multiple De- 
livery Systems: Some Lessons 
Learned." Paper presented at the In- 
diana Postsecondary Education Con- 
ference, Muncie, IN, October 25, 
1999. (ED 435 800) 

Describes how Indiana State Univer- 
sity offers courses for bachelor’s and 
master’s degrees in human resources 
development, technology education, 

‘ and vocational-technical education 
using satellite transmission to remote 
sites, videotape, and the Internet. 
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and Watt 1999, n.p.)* As King (1998) 
notes, “no existing technology can 
ensure academic honesty" (p. 27). 
However, some suggested security mea- 
sures include the following (Dicldnson 
1997; Gray 1997; Jarmon 1999;l<jng 
1998; Sephton 1999; Zirklc and 
Ourand 1999): 

• Holding proctored examinations 
at remote sites 

• Minimizing objective tests and 
focusing on methods calling for 
analysis and application (essays, 
case studies, etc.) 

• Constraining access to the 
website of a test through pass- 
words and usernames 

• Embedding quizzes into exercises 
and readings 

• Randomizing questions and the 
order in whid\ they arc presented 

• Carefully designing Web-based 
instruaion to convey outcomes 
the learner perceives as useful 
and desirable 

• Considering online exams in the 
same way as take-home tests. As 
Nelson (1998) points out, “they 
will never be asked on the job to 
sit away from all reference materi- 
als, forced to solve problems and 
communicate in isolation" (p. 2) 

• Using multiple sources of assess- 
ment data 

According to Gray (1997), the poten- 
tial of the Web to acatc a virtual com- 
munity can give instructors addi- 
tional “reality checks"; Through e- 
mail and web-based discussions, in- 
structors become familiar with the 
writing styles and abilities of indi- 
vidual students, so that an exam writ- 
ten in a radically different style would 
be a red flag. “Integrating collabora- 
tive elements such as joint projects, 
synchronous and asynchronous dis- 
cussion forums, and workgroup com- 
puting can further reduce the possi- 
bility of misrepresentation and in- 
crease student accountability" (ibid., 
n.p.). 

Programming 

Instructors’ lack of programming skill 
„ as well as the time involved can be 



major barriers to online assessment. 
Some commercial software allows 
customization without requiring pro- 
gramming, but it is often platform spe- 
cific or requires certain applications 
or players (Gray 1 999). Web editing 
software such as HotMctal or 
Frontpage can be used to create ITTML 
forms and tests. A number of resources 
listed on page 10 provide free ex- 
amples of JavaScript and Common 
Gateway Interface (CGI) programs, 
and the articles in the bibliography 
by Dickin,sori (1997), Doulai and 
Stace (1998), and Gray (1999) give 
advice for incorporating these tools in 
online assessment. Hayes and 
Huckstadt (2000) describe the use of 
Java scriptlets and RontPage to cre- 
ate online continuing education mod- 
ules for nurse practitioner self-study. 
Examples are provided at http:// 
WWW. nursing, twsu.edu/clp. 

Using the Medium 
to Advantage 

Online assessment offers a number of 
advantages (“Assessing Students" 

1 999; Juchnowski and Atkins 1 999; 
Ravitz 1998): 

• Ease of distribution 

• Timeliness (when the assessment 
is available and how long it is 
available) 

• Provision of fccdback—scores 
and explanations may be given 
immediately; e-mail allows for 
both individual and group 
feedback; learner feedback to the 
instructor can be incorporated 
immediately into the course 

• Links to tracking and manage- 
ment systems 

• Interactivity (sec the article on 
page 2) 

“If V^^WW-based learning systems sim- 
ply mirror the pragmatic weaknesses 
in the assessment process practiced by 
educators in traditional settings, there 
will be a lack of emphasis on the spe- 
cial advantages that the computer can 
provide" (Hopper 1998, p. 332). As- 
sessment at a distance should take 
advantage of the best features of the 



online environment. Nelson (1998) 
describes the Web’s potential for real- 
world, authentic perfonnance assess- 
ments. Web-based assessment enables 
true scaffolding (Hazari and Schnorr 
1 999); for example, an instructor can 
develop critical thinking questions 
based on the content and tenor of 
online discussion, rather than having 
a predetermined set of questions. Tlte 
online environment enables learners 
to regulate and monitor their own 
learning (Hazari and Schnorr 1999: 
Ravitz 1998). Informal quizzes can 
give learners instant feedback to gauge 
their understanding of a topic; 
hyperlinks can lead learners to sources 
that explain incorrect answers. "With 
its flexible participation structures, 
the Internet allows learners to pace 
their own activities, share information 
at opportune times, receive feedback 
and interact with others, all while 
building a permanent record of their 
activities and what they have learned” 
(Ravitz 1998, n.p.). 

The unique charaaeristia of the Web 
as a hypermedium give online learn- 
ing environments the potential to fa- 
cilitate dialogue — between learners 
and materials, their instructors, and 
fellow learners (Wild and Omari 
1996). The "learning community" 
model enables learners to conduct 
collaborative projects, engage in team- 
work. and adopt a variety of roles — 
all aspects of the contemporary work- 
place (Marshall 2000). Assessments 
of qualitative aspects such as coher- 
ence and comprehensiveness of 
projects, team funaioning, and inter- 
personal skills can thus reflect how 
learners will be evaluated in the world 
of work. 

Integration of content, online tasks, 
and assessment is an essential element. 
Hedberg and Corrent-Agostinho 
(2000) describe how Web-based group 
project tasks were completed at vari- 
ous intervals during a eburse rather 
than at the end of semester. The tasks 
thus became resources that learners 
accessed to assist them with the final 
group projea. 



[llMtence Learning 
Ammmmmm^nt Principles 



Innovations in Distance Education is a Penn State initiative that developed a set of principles 

and practices to guide the effective use of distance education. The essential principles for assess- 
ment of online learning, with examples of ways to put them into practice, are as follows (PSU 
1998): 

1 . Asscssmait instruments and activities should be congruent with the Icanung goals and skills 
required of the learner throughout a distance education program or course. 

Representative Practices 

a. Design the instmctional activities in the same form and methods as will be used to 
measure mastery of the program goals. 

b. At the start of the program, communicate the planned assessment and measurement 
strategies to the learner. Clearly state the nature, duration, due date, and impact on 
program grade of all assignments and measurement techniques. 

c. Where possible, provide assessment and measurement techniques and options that capi- 
talize on the unique characteristics and situations of the distance learner. 

2. Assessment a?jd measurement strategies should be integral parts of the learning experience, enabling 
learners to assess their progress, to identify areas for review, and to reestablish immediate learning 
or lesson goals. 

Representative Practices 

a. Consider a variety of “low-stakes" assessment and measurement strategies that enable 
learners to gauge their progress without affecting course grade or performance measure- 
ment. 

b. Create automated systems (c.g., online quiz tools, database programs) that can provide 
immediate feedback, relevant suggestions, and guided support. 

c. Develop techniques and systems that support leamcr-to-lcarner interactions for assess- 
ment and measurement. 

d. Design as many self-check activities as possible within the distance education program, 
enabling students to adjust their progress within the course. 

c. Consider credit for student effort as one component in the assessment process when the 
skill being taught is complex. 

3. Aswsswir/if and measurement strategies should accommodate the special needs, characteristics, and 
situations of the distance leanter. 

Representative Practices 

a. Consider the use of synchronous technologies such as teleconferencing or interactive com- 
pressed video for providing learners immediate assessment on work in progress. 

b. Consider the use of asynchronous technologies such as e-maii, bulletin boards, voice mail, 
fax, and/or other technologies to support assessment and measurement activities. 

c. Select media carefully when planning an assessment, recognizing that the use of several 
different media might be appropriate. For example, e-mail might be used to have students 
describe, online chats or telephone interviews might be used to have students discuss, and 
videotape might be used to demonstrate oral presentation skills or physical interactions, 
such as a counseling session. 

d. Where possible, use assessment and measurement strategies that use resources local to the 
distance learner. For example, use projects based on learners’ employment as part of the 
assessment and measurement for the program. 

e. If needed, help students make local arrangements to complete course assessments and 
evaluations. 

Reprinted with permission from the Mamgement Team for Innovations in Distance Education, Tenn State Universi^ 






Web Resaurtes 



Assessing Students Online, http:// 
vvvvw. ncvvcastlc.cdu.au/dcpartmcnt/ 
so/assess. htm. This site discusses stu- 
dent-centered learnirtg; the function 
of assessment; assessment outcomes, 
mechanisms, and planning; advan- 
tages and disadvantages of online as- 
sessment; and when it may not be suit- 
able. It also incUidcs links to programs 
that can be used to acate online tests 
and tutorials. 

Assessment/Testing/Evaluation Is- 
sues (Module 5 -Evaluation), http:// 
onlinc.parkJand.cc.il.us/ofh/Asscss- 
menuTviod5Issucs.htm .This clieckJist 
of issue.s that should be considered 
when assessing online is part of the 
Online Faculty Handbook from 
Parkland College’s Department of 
Virtual and Distance Learning. 

On-Line Evaluation: Multiple 
Choice, Discussion Questions, Es- 
say, and Authentic Projects, http^/ 
wind.cc.whccn.edu/~gnelson/papcr/ 
TCC-98.htini. Describes an adapta- 
tion of traditional assessment to the 
online environment with specific ex- 
amples in earth sciences. Emphasis is 
on authentic assessment of students' 
communications skills, problem-solv- 
ing ability, and teamwork. 

RMIT University School of Busi- 
ness, Melbourne, Australia, http:// 
ww\\'. bf. rmit.edu ,au/ve t/man/i n f40 1 / 
html/assessmcnt.himl. This model of 
online assessment shows how to 
present assessment tasks and criteria 
as well as sequence the learner path- 
way. Includes task descriptions, links 
to more information, assessment cri- 
teria, deadlines, and the weightings 
of the different tasks. 

Tcx)ls and Techniques 

A Beginner's Introduction to On- 
Line Web-Based Testing: http:// 
www.unb.ca/web/econ/scphton/test- 
ing/. Discusses the desirable features 
of online tests and software and of- 
fers suggestions for developing pro- 
grams and getting free ones. Includes 



the nitty-gritty of getting testing 
websites to work. Provides links to do- 
it-yourself sites on )ava, JavaScript, 
Peri, and C programming. 

Creating Multimedia/Interactivc 
7csts for the Internet in 10 Minutes, 
h ttpy/www. al n . org/alnwcb/magazi ne/ 
is$ue2/rob.htm. An article from Asyn- 
chronous Learning Networks Maga- 
zine describes QuizPlease software, 
which is designed to offer tutoring as 
well as testing. 

Online Educational Delivery Appli- 
cations: A Wcbtool for Comparative 
Ana ly si s . h ttp ://www, cit.be . ca/ 
landonlinc/ or http://www.olin.nf.ca/' 
landonline/. Designed to help educa- 
tors evaluate and select online deliv- 
ery software. Describes and compares 
the most viable applications in use, 
focusing on technical specifications, 
instructional design values, media ca- 
pabilities, tools, ease of use, potential 
for collaboration and connectivity, 
and contact information. 

QuizCODE— A Tool for Online As- 
sessment and Feedback, http:// 
www.enablc.cvitech.fi/enable99/ 
papers/quizcodc/qui/code.htrnl. De- 
scribes QuizCODE XiML-bascd assess- 
ment software. Includes examples of 
future uses such as delivering quizzes 
over personal data assistants and mo- 
bile phones. 

Rubrics for Web Lessons, http:// 
edweb.sdsu.edu/triton/July/rubrics/ 
Rubrics_forJvVcb_Lcssons.html. TTiis 
site discusses authentic assessment 
and the use of nibrics with examples, 
templates, and links to further read- 
ing. 

The CASTLE (Computer ASsisted 
Teaching and LEarning) Toolkit. 
http://www.le.ac.uk/cc/ltg/castle/ 
index.html. These tools make it pos- 
sible to create and manage online in- 
teractive multiple choice questions. 
Includes tips and good practice for 
question and test design. 
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The Design, Development and De- 
livery of Internet Based Training and 
Education, lKtp;//tcleeducation. 
nb.ca/mcdia/03.2000/ddd-ibte/ 
index.html. Contains /\n Inventory of 
Design, Development, and Delivery 
Software for Intcmct-Bascd Learning, 
a comparison of 33 products such as 
QuestionMark, 'Toolbook, and 
FirstClass. 

Website Building Tools 

CGI Script Archive: bignosebird.com/ 
carchive^nbform.shtml 

Matt's Free Perl CGI Scripts: http:// 
www.worldwidcmuit.com/scripts/ 

Page Tutor: httpy/www. pagetutor.com. 
Free HTML tutorials; tables, frames, 
and forms tutorials; a MouscOver tu- 
torial; a javascript tutorial; password 
protection script; and lots of resources 
for beginning and advanced HTML 
authors and web designers. 

Conferencing Tools 

Conferencing Software for the Web: 
http://thirtkofit.com/webconf/. A 
guide to Web software (free and com- 
mercial) for text-based, asynchronous 
group discussions, maintained by 
David R. Woolley. 

Meeting Web: wvw.meetingweb.com 
Conferencing software 

Toolbook IJ: httpvyhome,click2learn. 
com/ (formerly Asymetrix) 

Assistant: c-Learning authoring pro- 
gram; Instructor: high-end authoring 
tool designed for creating e-Leaming 
courses and applications 

Testing Tools 

Online Exercises System: http:// 
math.uc du/onex/demo. html 
Question Mark: http;//ww^v.qmark. 
com/ 

Self-Test: www.usq.edu.au/uscrs/ 
tay lorj a/Se 1 ftest. h tm 
WebTest: http://fpg.uwaterloo.ca/ 
WEBTEST/WEBTEST_intro.html 
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